Achieving high inter-rater reliability in establishing data labels: a retrospective chart review study.

BACKGROUND: In medical research, the effectiveness of machine learning algorithms depends heavily on the accuracy of labeled data. This study aimed to assess inter-rater reliability (IRR) in a retrospective electronic medical chart review to create high quality labeled data on comorbidities and adverse events (AEs). METHODS: Six registered nurses with diverse clinical backgrounds reviewed patient charts, extracted data on 20 predefined comorbidities and 18 AEs. All reviewers underwent four iterative rounds of training aimed to enhance accuracy and foster consensus. Periodic monitoring was conducted at the beginning, middle, and end of the testing phase to ensure data quality. Weighted Kappa coefficients were calculated with their associated 95% confidence intervals (CIs). RESULTS: Seventy patient charts were reviewed. The overall agreement, measured by Conger's Kappa, was 0.80 (95% CI: 0.78-0.82). IRR scores remained consistently high (ranging from 0.70 to 0.87) throughout each phase. CONCLUSION: Our study suggests the detailed manual for chart review and structured training regimen resulted in a consistently high level of agreement among our reviewers during the chart review process. This establishes a robust foundation for generating high-quality labeled data, thereby enhancing the potential for developing accurate machine learning algorithms.

BERT-Based Neural Network for Inpatient Fall Detection From Electronic Medical Records: Retrospective Cohort Study.

BACKGROUND: Inpatient falls are a substantial concern for health care providers and are associated with negative outcomes for patients. Automated detection of falls using machine learning (ML) algorithms may aid in improving patient safety and reducing the occurrence of falls. OBJECTIVE: This study aims to develop and evaluate an ML algorithm for inpatient fall detection using multidisciplinary progress record notes and a pretrained Bidirectional Encoder Representation from Transformers (BERT) language model. METHODS: A cohort of 4323 adult patients admitted to 3 acute care hospitals in Calgary, Alberta, Canada from 2016 to 2021 were randomly sampled. Trained reviewers determined falls from patient charts, which were linked to electronic medical records and administrative data. The BERT-based language model was pretrained on clinical notes, and a fall detection algorithm was developed based on a neural network binary classification architecture. RESULTS: To address various use scenarios, we developed 3 different Alberta hospital notes-specific BERT models: a high sensitivity model (sensitivity 97.7, IQR 87.7-99.9), a high positive predictive value model (positive predictive value 85.7, IQR 57.2-98.2), and the high F1-score model (F1=64.4). Our proposed method outperformed 3 classical ML algorithms and an International Classification of Diseases code-based algorithm for fall detection, showing its potential for improved performance in diverse clinical settings. CONCLUSIONS: The developed algorithm provides an automated and accurate method for inpatient fall detection using multidisciplinary progress record notes and a pretrained BERT language model. This method could be implemented in clinical practice to improve patient safety and reduce the occurrence of falls in hospitals.

Use of Epic Electronic Health Record System for Health Care Research: Scoping Review.

BACKGROUND: Electronic health records (EHRs) enable health data exchange across interconnected systems from varied settings. Epic is among the 5 leading EHR providers and is the most adopted EHR system across the globe. Despite its global reach, there is a gap in the literature detailing how EHR systems such as Epic have been used for health care research. OBJECTIVE: The objective of this scoping review is to synthesize the available literature on use cases of the Epic EHR for research in various areas of clinical and health sciences. METHODS: We used established scoping review methods and searched 9 major information repositories, including databases and gray literature sources. To categorize the research data, we developed detailed criteria for 5 major research domains to present the results. RESULTS: We present a comprehensive picture of the method types in 5 research domains. A total of 4669 articles were screened by 2 independent reviewers at each stage, while 206 articles were abstracted. Most studies were from the United States, with a sharp increase in volume from the year 2015 onwards. Most articles focused on clinical care, health services research and clinical decision support. Among research designs, most studies used longitudinal designs, followed by interventional studies implemented at single sites in adult populations. Important facilitators and barriers to the use of Epic and EHRs in general were identified. Important lessons to the use of Epic and other EHRs for research purposes were also synthesized. CONCLUSIONS: The Epic EHR provides a wide variety of functions that are helpful toward research in several domains, including clinical and population health, quality improvement, and the development of clinical decision support tools. As Epic is reported to be the most globally adopted EHR, researchers can take advantage of its various system features, including pooled data, integration of modules and developing decision support tools. Such research opportunities afforded by the system can contribute to improving quality of care, building health system efficiencies, and conducting population-level studies. Although this review is limited to the Epic EHR system, the larger lessons are generalizable to other EHRs.